Visual Probing: Cognitive Framework for Explaining Self-Supervised Image Representations

Recently introduced self-supervised methods for image representation learning provide on par or superior results to their fully supervised competitors, yet the corresponding efforts explain approaches lag behind. Motivated by this observation, we introduce a novel visual probing framework explaining models leveraging tasks employed previously in natural language processing. The require knowledge about semantic relationships between parts. Hence, propose systematic approach obtain analogs of vision, such as words, context, and taxonomy. Our proposal is grounded Marr’s computational theory vision concerns features like textures, shapes, lines. We show effectiveness applicability those context representations. key findings emphasize that relations can serve an effective intuitive tool discovering how machine work, independently data modality. work opens plethora research pathways towards more explainable transparent AI.